Skip to content

[Kernel] Integrate CUTLASS MoE kernel with PPLX #18762

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 25 commits into from
Jun 7, 2025
Merged

[Kernel] Integrate CUTLASS MoE kernel with PPLX #18762

merged 25 commits into from
Jun 7, 2025

Conversation

@ElizaWszola
Copy link
Contributor

@ElizaWszola ElizaWszola commented May 27, 2025 •
edited by github-actions bot
Loading

Integrate CUTLASS MoE fp8 kernels with PPLX.

Unit tests:

tests/kernels/moe/test_pplx_cutlass_moe.py

E2E testing:

export MASTER_ADDR=127.0.0.1
export MASTER_PORT=29500
export VLLM_ALL2ALL_BACKEND=pplx
python3 examples/offline_inference/data_parallel.py \
        --model="nm-testing/DeepSeek-Coder-V2-Lite-Instruct-FP8" \
        --dp-size=2 \
        --tp-size=1 \
        --trust-remote-code

@ElizaWszola
Cutlass MoE pplx - working unit tests
888177a 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Working e2e, but there are some hacks and it needs cleaning
6486345 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Set the correct workspace shapes, padded and unpadded c1,c2,c3
5d4751f 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
format
9cb7802 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Merge branch 'main' into cutlass-moe-pplx-integration
f34d6b1 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
uncomment quant_method selection
9499f74 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Working e2e after merge
df1a014 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@github-actions
Copy link

github-actions bot commented May 27, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@ElizaWszola ElizaWszola changed the title [Kernel] Integrate CUTLASS MoE kernel with PPLX [WIP][Kernel] Integrate CUTLASS MoE kernel with PPLX May 27, 2025
@ElizaWszola
Nuke output map codepath, clean up a bit
503a9b3 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Fix the non-pplx codepath
8c1d57b 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
CUDA kernel for pplx data computation, cleanups, fixing unit tests an…
268bbea 
…d benchmarks

Signed-off-by: ElizaWszola <ewszola@redhat.com>
@mergify mergify bot added the ci/build label May 28, 2025
@ElizaWszola
Various cleanups
a6236bf 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Better types and attribute checks
ab46919 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Missing return, type check ignore
3e436fd 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola ElizaWszola changed the title [WIP][Kernel] Integrate CUTLASS MoE kernel with PPLX [Kernel] Integrate CUTLASS MoE kernel with PPLX May 28, 2025
@ElizaWszola ElizaWszola marked this pull request as ready for review May 28, 2025 16:10
bnellnm
bnellnm reviewed May 28, 2025
View reviewed changes
bnellnm
bnellnm reviewed May 28, 2025
View reviewed changes
bnellnm
bnellnm reviewed May 28, 2025
View reviewed changes
bnellnm
bnellnm reviewed May 28, 2025
View reviewed changes
bnellnm
bnellnm reviewed May 28, 2025
View reviewed changes
bnellnm
bnellnm reviewed May 28, 2025
View reviewed changes
tlrmchlsmth
tlrmchlsmth reviewed May 28, 2025
View reviewed changes
Copy link
Member

@tlrmchlsmth tlrmchlsmth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments, but looks good overall -- lets try to get it landed once those and Bill's comments are addressed!

@mergify mergify bot removed the needs-rebase label Jun 3, 2025
@mergify
Copy link

mergify bot commented Jun 3, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ElizaWszola.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Jun 3, 2025
@ElizaWszola
Merge branch 'main' into cutlass-moe-pplx-integration
1cb0fb6 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@mergify mergify bot removed the needs-rebase label Jun 4, 2025
tlrmchlsmth
tlrmchlsmth reviewed Jun 4, 2025
View reviewed changes
@ElizaWszola
Comment about different codepaths, is_pplz variable
ec55bee 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Merge branch 'main' into cutlass-moe-pplx-integration
f5b4ff0 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
fix workspace_shapes args
f6e968d 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@mergify
Copy link

mergify bot commented Jun 5, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ElizaWszola.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Jun 5, 2025
@tlrmchlsmth
Merge branch 'main' into cutlass-moe-pplx-integration
24fa6cb 
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
@tlrmchlsmth tlrmchlsmth added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 5, 2025
@tlrmchlsmth tlrmchlsmth enabled auto-merge (squash) June 5, 2025 17:37
@mergify mergify bot removed the needs-rebase label Jun 5, 2025
@ElizaWszola
Merge branch 'main' into cutlass-moe-pplx-integration
ba6ce8f 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
@ElizaWszola
Import fixes, headers
db5aa2d 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
auto-merge was automatically disabled June 6, 2025 04:48

Head branch was pushed to by a user without write access

@aarnphm aarnphm enabled auto-merge (squash) June 6, 2025 17:43
@simon-mo simon-mo disabled auto-merge June 7, 2025 01:26
@simon-mo simon-mo merged commit 84166fe into vllm-project:main Jun 7, 2025
93 of 96 checks passed
bnellnm added a commit to neuralmagic/vllm that referenced this pull request Jun 11, 2025
@bnellnm
Revert "[Kernel] Integrate CUTLASS MoE kernel with PPLX (vllm-project…
bad6cb5 
…#18762)"

This reverts commit 84166fe.

Signed-off-by: Bill Nell <bnell@redhat.com>
@yeqcharlotte yeqcharlotte mentioned this pull request Jun 16, 2025
Closed
This was referenced Jun 24, 2025
Closed
Merged
@bennorris123
Copy link

bennorris123 commented Aug 27, 2025 •
edited
Loading

Hi @ElizaWszola - Is the pplx kernel already in the vllm image by default now? Or do we still need to go through the additional installation steps of expert parallelism (https://github.com/vllm-project/vllm/tree/main/tools/ep_kernels). Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tlrmchlsmth tlrmchlsmth tlrmchlsmth approved these changes

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

+1 more reviewer

@bnellnm bnellnm bnellnm approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ElizaWszola @bennorris123 @tlrmchlsmth @bnellnm @simon-mo